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(54) Device and system for labeling sight images 

(57) The device and system for sight labeling ac- 
cording to the present invention comprises an image ac- 
quiring section for taking images, a position in formation 
acquiling section for recording a camera position while 
the images are being taken, camera descriptive infor- 
mation acquiling section for acquiring the camera angle, 
focal distance and image size while the image are being 
taken, a map information management section for man- 
aging map information, determining a view space inside 
map information space based on the acquired position, 



camera angle, focal distance and image size, and cap- 
turing constructions present inside the view space, a la- 
bel information preparation section for preparing label 
information including the names or descriptive informa- 
tion of constructions and transfer positions, a labeling 
information output section for overlaying the names or 
descriptive information from the map information at po- 
sitions in the images corresponding to position informa- 
tion in the prepared label information, and a control sec- 
tion for controlling each of the sections described above. 
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Description 

BACKGROUND OF THE INVENTION 
Field of the Invention 



20 



25 



Conventional Art 

As convention, systems to, noting users of 
are various types of navigation systems. F"^ 9 ',^^^^^ the position of vehicle by referring 
discloses a navigation device comprising a posrtion renewa sector, o g p generating section 

to road map data when position data and movement ^W^^*™*^™ p y , jke a 9 , hree<lim ensional 
for generating disp.ay road data and display ^^^^^S^^Lea on these display data, 
video image data generating section for prepar ,ng ^^^^^^ and transit areas, this navigation 

S=h^^^ 

the user is actually traveling. Hov/ir^ th«=> user must ultimately compare the actual 

However, when performing route 9^,^"^^^ Jj, £ ™ a " the objects'n the actual scenery 
scenery with the geographical information ^^^^^^^^Winihe user' s perspective is, the user 
are. In other words, in order to identify what a bu.ld.ng. road or ^J'"^^ sees and the 

must subconsciously work the brain in order to map prepared by the 

so symbols in the map which is displayed as a moving image. In crt.es the ' u £^™^ recognjzi ng 
computer with the actua. scenery in order to grasp the user s " ^ h JJ^* ulldlng . For this reason, 

the features of buildings in the direction of P^ress-oarecon iders the map to den,, Mhe 9 ^ ^ ^ 

■ retse 0 ! 2K 2SJ — is difficult to see. and ,t is 

35 difficult to make a correlation. 

SUMMARY OF THE INVENTION 

The present invention has the object o, offering a device and system ^^^^^^^^ 

° f ^^-'^^ invention is a device,. ^^^TZ ^i^X 
for taking images: a position information acquiring sect™ tor ^ u,n "9 a ^^ a ^ 3 focal distance and an image 
taken: a camera descriptive information acquiring section torw«nga J* 8 ^^ ™ Wormalion . deter- 

size while the images are being taken: a map ^^^^^^^^J^. focal distance and 
mining a view space inside a map information space ^^^^^^^^ preparation section for 

and a sight labe.ing center: the sigh, labeling terminal ^^^^ taken, a camera 



40 



45 
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angle., the focal distance and the image size through the communication network to the sight labeling center and 
receiving label information. from the sight labeling center, a label information output section for overlaying names or 
descriptive information of constructions inside the label information at corresponding positions in the images, and 
outputting the overlayed images to a visual device, and a terminal control section for controlling the sections described 

5 above; and the sight labeling center comprising a communication control section for receiving information relating to 
division of the images into sub-domains, the camera angle, the focal distance and the image size through the commu- 
nication network from the sight labeling terminal, and sending the label information to the sight labeling terminal, a map 
information management section for managing map information, determining a view space inside map information 
space based on the received position, camera angle, focal distance and image size, and capturing constructions 

w present inside the view space, a label information preparation section for correlating the captured constructions with 
respect to the sub-domains of the images : and preparing label information including the names or descriptive informa- 
tion of the correlated constructions and their transfer positions, and a center control section for controlling the sections 
described above. 

Another embodiment of the present invention is a device for sight labeling comprising an image acquiring section 

is for taking images: a position information acquiring section for acquiring a camera position while the images arc being 
taken; a camera descriptive information acquiring section for acquiring a camera angle, a focal distance and .-in image 
size while the images are being taken: an image processing section for dividing images acquired by the image acquiring 
section into a plurality of sub-domains: map information management section for managing map infonnahon deter- 
mining a view space inside a map information space based on the acquired position, camera angle, foe*! distance and 

20 image size : and capturing constructions present inside the view space: a label information preparation section for 
correlating the captured constructions with respect to sub-domains of the images, and preparing label =r.!crm;-inon 
including the names or descriptive information of the correlated constructions and their transfer positions i commu- 
nication address memory section for storing communication addresses of communication devices relating !o mo con 
structions: a communication processing section for setting up communication paths with the communication , ac!t esses 

25 based on the received communication addresses: a label information output section for overlaying the narro* oi de- 
scriptive information from the map information at positions in the images corresponding to the transfer positions < fl the 
prepared label information, and outputting the overlayed images'to a visual device: a screen position indicting so :; on 
for allowing a user to indicate a position on the screen of the visual device, determining the construction r ■ :f -e :t;oi 
information corresponding to the indicated screen position when a position on the screen of the visual device s --.i.c iiod 

30 by the user, determining the communication address of the communication device relating to the determine--:: ■ ■ : ■■ ^ 
tion from the communication address memory section: and a control section for controlling the sections .:• - ::ea 
above. 

Another embodiment of the present invention is a system for sight labeling comprising a sight labci-:* ; . 
and a sight labeling center: the sight labeling terminal comprising an image acquiring section for takm- ■•■»;■■- 1 

35 position information acquiring section for acquiring a camera position while the images are being take- 

descriptive information acquiring section for acquiring a camera angle, a focal distance and an image s / v 
images are being taken, an image processing section for dividing the taken images into a plurality of sui: = 
communication control section for sending information relating to division of the images into sub-domains 
angle, the focal distance and the image size through the communication network to the sight label" 

40 receiving label information and communication addresses from the sight labeling center, a communicate; - : 
section for setting up paths to the communication addresses based on the received communication addi e 
information output section for overlaying names or descriptive information of constructions inside the lab- 
at corresponding positions in the images, and displaying the overlayed images on a visual device, a s. • 
indicating section for allowing a user to indicate a position on the screen of the visual device, determining 

45 tion in the label information corresponding to the indicated screen position when a position on the scree: 
device is indicated by the user outputting the construction to the communication control section, at- : 
communication address of the communication device relating to the construction received by the commit 
section to the communication processing section: and a terminal control section for controlling the sec: ■ 
above: and the sight labeling center comprising a communication control section for receiving inform.:' 

so division of the images into sub-domains, the camera angle, the focal distance and the image size throu :■ 
nication network from the sight labeling terminal, and sending the label information and communication - 
sight labeling terminal, a map information management section for managing map information, dote--- 
space inside map information space based on the received position, camera angle, focal distance and •• • . 
capturing constructions present inside the view space, a label information preparation section for com 

55 tured constructions with respect to the sub-domains of the images, and preparing label information inc k. : : 
or descriptive information of the correlated constructions and their transfer positions, a communication . •: : 
section for storing communication addresses of communication devices relating to the constructions = 
communication addresses of constructions indicated by information of constructions in the label in! 
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center control section for controlling the sections described above. ^ minSk i 
Another embodiment of the present invention is a system for sight labeling comprising a sight labeling terminal 
and a sight labeling center: the sight labeling terminal comprising an image acquiring section for taking images, a 
position information acquiring section for acquiring a camera position while the images are being taken, a camera 
descriptive information acquiring section for acquiring a camera angle, a focal distance and an image size while the 
images are being taken, an image processing section for dividing the taken images into a plurality of sub-domains, a 
communication control section for sending information relating to division of the images into sub«tomains. the camera 
angle the focal distance and the image size through the communication network to the sight labeling center and 
receiving label information and communication addresses from the sight labeling center a label information output 
section for overlaying names or descriptive information of constructions inside the label information at corresponding 
positions in the images, and displaying the overlayed images on a visual device, and for displaying » communication 
address on the visual device when the communication address is received, a screen position md.cni.ng scct.on to 
allowing a user to indicate a position on the screen of the visual device, determining the construct ,n the label 
information corresponding to the indicated screen position when a position on the screen of the visual device .s .ndicated 
by the user outputting the construction to the communication control section, and sending the commun.CHi.on address 
of the communication device relating to the construction received by the communication control secnon to the labe 
information output section: and a terminal control section for controlling the sections described above <,nd he sight 
labeling center comprising a communication control section. for receiving information relating to division o, the .m^ges 
into sub-domains, the camera angle, the focal distance and the image size through the communicai.on nct.^k from 
the sight labeling terminal, and sending the label information and communication address to the sight Ihooi.^ •-m.ivii 
a map information management section for managing map information, determining a view space ins.de n,,:: ^ or mo- 
tion space based on the received position, camera angle, focal distance and image size, and capturing ccr,- ,ci.ons 
present inside the view space, a label information preparation section for correlating the captured const. l. 
respect to the sub-domains of the images, and preparing label information including the names or descr.pt* 
tion of the correlated constructions and their transfer positions, a communication memory section for sio.u 

nication addresses of communication devices relating to the constructions, and outputting commun.CHt.or> 

of constructions indicated by information of constructions in the label information, and a center coni.o. 

controlling the sections described above. 

Since the present invention correlates map information in a computer with each part of a sight .m*, 

actual scenery and displays the results to a user, humans are not required to make correlations by comp, 

on a computer with the actual scenery. 

Additionally since another embodiment of the present invention correlates map information innu- 

each part of a sight image of the actual scenery and displays the results to a user, humans are not requ. 

correlations by comparing a map on a computer with the actual scenery, and since it is also possible, s.mp.y : 

an area of the screen, to be notified of the telephone numbers or the like of residents of the building o 

area or to automatically make telephone calls or the like thereto, it can be used by users as an extreme v 

new type of telephone directory. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a diagram showing the structure of a sight labeling device according to a first embodiment 
invention. 

Fig. 2 is a diagram showing the structure of a sight labeling device according to a second emb- ■■: 

present invention. . * ^ . 

Fig. 3 is a flow chart showing the process carried out by the sight labeling device of the second . 

the present invention. 

Fig. 4 is a diagram showing the data structure of sight image file. 
Fig. 5A is an example of two-dimensional map. 
Fig. 5B is an example of a three-dimensional map. 
Fig 6 is a diagram showing a method of calculating a view space. 

Fig. 7 is a diagram showing an example of a view space in a three-dimensional map space. 
Fig. 8 is a diagram showing an example of a projection. 

Fig. 9 is a diagram showing an example of the division of sub-domains in a sight image. 
Fig 10 is a diagram showing the division of sub-domains in a CG image. 

Fig. 11 is a diagram for explaining pattern matching of sub-domains of a sight image with the - .: 
CG image. 

Fig 1 2 is a diagram showing an example of overlaying label information on a sight image. 
Fig. 13 is a diagram showing the structure of a sight labeling system of the present invention 
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Fig. 1 4 is a diagram showing the structure of a sight labeling device of a third embodiment of the present invention. 
Fig. 15 is a flow chart showing the process canied out by the sight labeling device of Fig. 14. 
Fig. 16 is a diagram showing the structure of a sight communication system of the present invention using the third 
embodiment. 

Fig. 1 7 is a diagram showing the structure of a distance-reference sight labeling device of a fourth embodiment of 
the present invention. 

Fig. 18 is a flow chart showing the process canied out by a sight labeling device of the embodiment of Fig. 17. 
Fig. 1 9A is an example of a two-dimensional map. 
Fig. 19B is an example of a three-dimensional map. 

Fig. 20 is a diagram showing the structure of a distance-reference sight labeling device of a fifth embodiment of 
the present invention. 

Fig. 21 is a diagram showing the structure of a sight labeling system of the present embodiment. 
DESCRIPTION OF PREFERRED EMBODIMENTS 

First, an example of the basic structure of a device for labeling sight images achieved by the present invention 
(hereinafter referred to as a sight labeling device) shall be explained. The sight labeling device of the present invention 
prepares map data in a computer in the form of three-dimensional data, obtains the camera position, camera angle, 
focal distance and image size of filmed image (hereinafter referred to as a "sight image" in order to discriminate it from 
a computer graphics (hereinafter, CG) image) during filming, determines a view space (visual space) for a case of 
viewing from the camera position and angle and focal distance in the three-dimensional map space in the computer 
obtains geographical information regarding constructions contained in the determined view space, then overlays and 
displays the geographical information on the sight image which is the actual scenery, thereby allowing for correlation 
therebetween. The sight image forming the background for overlaying and displaying the geographical information is 
not necessarily restricted to being images of a filmed sight image displayed on a monitor and may for example be 
images of actual scenery transmitted through glass such as a windshield or reflected by a mirror or the like. This 
geographical information includes the names and/or descriptive information relating to constructions and the like pre- 
pared in the computer, descriptive information meaning information regarding the various characteristics (such as 
profile and color) of constructions. The term "constructions" as referred to in the present specification is used not only 
to refer to manmade constructions, but also to all data having any kind of geographical structure in the map database, 
including mountains, rivers and oceans. When geographical information is overlayed and displayed on a sight image, 
the position in the sight image (hereinafter referred to as the "transfer position") at which each construction indicated 
by the geographical information should appear is determined based on the camera position, camera angle, focal dis- 
tance and image size, in order to overlay and display the name or descriptive information of the construction. 

Furthermore, for the purposes of further increasing the level of precision of correlation between constructions in 
the sight image and constructions in the CG image, the constructions first captured in each sub<Jomain of the sight 
image are correlated by pattern matching. The CG image is prepared based on the captured constructions, the sub- 
domains of the CG image are correlated to sub-domains in the sight image by pattern matching, and the construction 
which is the basis of the correlated sub-domain is determined. Hereinbelow, an example of a method for preparing CG 
images shall be described. The three-dimensional map database (hereinafter referred to as M DB") is accessed on the 
basis of the previously obtained camera position, camera angle, focal distance and image size to determine the view 
space in the three-dimensional map space. The constructions in the view space are determined and the three-dimen- 
sional data for each construction is transformed into a three-dimensional projection onto a projection plane formed by 
the camera screen. Furthermore, of the line data forming the projection diagram of each construction, the line data 
which are not capable of being seen due to being occluded by other constructions are eliminated as occluded lines by 
using a method such as a normal vector method. The CG image is divided into sub-domains based on the line data 
which remains after elimination of occluded lines. Since a three-dimensional map data DB is being used, the name of 
the construction corresponding to each sub-domain is correlated to that sub-domain. 

Then, the name of the construction in each sub-domain of the CG image correlated to each sub-domain of the 
sight image by means of pattern matching is extracted. The positional coordinates of the actual scenery image onto 
which the extracted construction name is to be overlayed are determined by three-dimensional projective transforma- 
tion of the positional coordinates of the construction in the three-dimensional map space onto the projection plane. 
Label information is prepared from the positional coordinates of the actual scenery image onto which the extracted 
construction name is to be overlayed. The construction name is overlayed onto the sight image which is the actual 
scenery based on the label information, and displayed by means of a view visual device. 

Furthermore, the sight labeling device according to the present invention may also be provided with communication 
address memory means for storing communication addresses of communication devices relating to the constructions 
(telephone number fax number. IP address, email address, WWW page address, etc.). screen position indication 
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means for indicating to the user a position on the screen of the visual device, and communication processing means 
for setting up a communication path with a communication address when the communicat.on address is recerved. In 
this case when the user indicates a position on the screen of the visual device, the screen position indication means 
determines the construction in the label information corresponding to the indicated screen position, receives thecom- 
s munication address of the communication device relating to the determined construction from the communication ad- 
dress memory means, and sends this to the communication processing means. The communicat.on processing means 
is capable of setting up a communication path with the communication address based on the received commun.ca .on 
address. As a result, the user is able to telephone, fax or email to residents in the construction of the label information 
simply by touching an area of the screen. 
10 In the above-described structure, the structure may be such that when a user indicates a position on the screen 

of the visual device, the communication processing means first displays the received communication address infor- 
mation on the screen of the visual device, and the communication path with the communication address .s set up 
depending on a communication start request of the user. Furthermore, it is possible to provide destination .nd.cation 
means for the user to indicate the destination, wherein the label information preparation means appends destmat.on 
is identifying information indicating that it is the destination to label information with respect to a destination indicated by 
the user so that the label information output means will display that a construction matches the destination conditions 
when destination identifying information is appended to label information. Furthermore, when a user indicates a pos. ion 
on the screen of visual device, a screen position indication.means determines the construction .n the label information 
corresponding to the designated screen position, receives the communication address of the communicat.on device 
20 relating to the determined construction from the communication address memory means, and sends the communication 
address to the label information output means, so that the label information output means can overlay and display the 
received communication address information on the sight image. 

In order to further raise the level of precision of correlation between constructions in the sight image and construc- 
tions in the CG image in the sight labeling device of the present invention, it is possible to use depth values, which are 
25 the distances from each point in the sight image to the camera position at the time the image was taken, for each point 
in the sight image. In the same manner, the depth values which are the distances from the points in each sub-doma.n 
of the CG image to the camera position at the time the image was taken are determined. For example, the rat.o of the 
average of a group of depth values in each sub-domain of the CG image to the average of a group of depth values .n 
each sub-domain of the sight image is determined. From this ratio, it is determined whether or not ,t * poss.ble to 
so correlate the sub-domain of the sight image with the sub-domain of the CG image. Alternatively, it is possible to deter- 
mine the ratio of overlap between the sub-domains in the sight image and the sub-doma.ns .n the CG image, and to 
decide whether or not it is possible to correlate the sub-domain of the sight image with the sub-domain of the CG image 
from both the ratio of average depth and the ratio of overlap between the subOomains. A CG image is prepared based 
on the captured constructions, the sub-domains in the CG image are correlated to sub-domains in the sight .mage by 
35 pattern matching, and the constructions on which the con-elated sub-domains are based are determined^ Of course^ 
the ratio is not necessarily restricted to being of average depth values, and may be any statistical value obta.ned from 

Snbetow^an embodiment of the present invention shall be explained in detail with reference to the drawings. 
Fig. 1 is a diagram showing the construction of a sight labeling terminal according to a first embod.ment of the 

40 present invention. . Hiatal 

The siqht labeling device of the present embodiment comprises an image acquiring section 1 such as a d g.ta 
camera for acquiring images, a position information acquiring section 2 such as a GPS (Global Position.ng System) 
Receiver for recording the position of the camera while images are being taken, a camera descriptive mformat.on 
acquiring section 3 such as a three-dimensional electronic compass attached to the digital camera for recording the 

45 camera angle, focal distance and image size while images are being taken, a map information management sect.on 
5 such as a map DB management program for managing map information, determining the view space in the map 
information space based on the recorded position, camera angle, focal distance and image size and capturing the 
constructions present in the view space, a label information preparation section 6A for preparing label informal on 
including the names or profiles and positions of constructions, a label information output section 7A for overlay.ng the 

so names or profiles in the map information at the positions in the images corresponding to the information regarding 
positions in the obtained label information and outputting the overlayed images to a visual device (not shown ,n the 
drawing) and a control section 8A for controlling the respective sections 1 -7A. 

Fig 2 is a diagram showing the structure of sight labeling device according to a second embodiment of the present 
invention and Fig. 3 is a flow chart showing the process carried out by the sight labeling device of F,g. 2. The second 

55 embodiment differs from the first embod.ment by having an image processing section 4 for processing images taKen 
by the image acquiring section 1 separate from the control section. Since the flow of the basic processes of the .ftrs 
embodiment shown in Fig. 1 and the second embodiment shown in Fig. 2 are identical, the operations of ^ embodiment 
shown in Fig 2 shall be explained in detail, while the operations of the embodiment shown in Fig. 1 shall be ommea. 
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The sight labeling device of the embodiment shown in Fig. 2 comprises an image acquiring section 1 such as a 
digital camera for acquiring sight images, a position information acquiring section 2 such as a G PS receiver for recording 
the camera position while the images are being taken, a camera descriptive information acquiring section 3 such as a 
three-dimensional compass attached to the digital camera for recording the camera angle, focal distance and image 

5 size while the images are being taken, an image processing section 4 for dividing the taken images into a plurality of 
sub-domains, a map information management section 5 for managing the map information, determining the view space 
in the map information space based on the recorded position, camera angle, focal distance and image size, and cap- 
turing the constructions present in the view space, a label information preparation section 6B for correlating the con- 
structions captured with respect to the sub-domains in the image by means of pattern matching and preparing label 

io information including the names or descriptive information and transfer positions of the correlated constructions, a 
label information output section 7 for overlaying the names or descriptive information of the constructions in the gen- 
erated label information at the corresponding positions in the image, a label information output section 7B for outputting 
the overlayed image to a visual device, and a control section 8 for controlling the respective sections 1 -7B. 

Next, the operations of the present embodiment shall be explained in detail with reference to Fig. 3. When the 

is sight labeling device is activated, the control section 8B first sends procedure initiation commands to the position 
information acquiring section 2, the camera descriptive information acquiring section 3 and the image acquiring section 
in order to obtain information regarding the sight image. Upon receiving instructions from the control section 8, the 
position information acquiring section collects position information from the GPS receiver or the like every second, and 
sends these to the control section 8B ^step 21 ). Here, the time interval is not necessarily restricted to units of seconds. 

20 Upon receiving instructions from the control section 8 : the image acquiring section 1 obtains sight images every second, 
and sends them to the control section 8B (step 22). Upon receiving the instructions from the control section 8, the 
camera descriptive information acquiring section 3 records the camera angle of the sight image acquiring device while 
the images are being taken as a pair of a horizontal angle and an elevation angle (step 23). Simultaneously if the sight 
image device has a zoom capability, the focal distance is also obtained (step 24). Since the image size is standard for 

25 • each sight image device, the control section 8B stores the image size information. The control section 8B stores the 
collected information as a sight image file. 

Fig. 4 shows an example of a file format which is a data construction for sight image files, the sight image file 
shown in Fig. 4 has header information and image data. The header information includes position information, camera 
angle information, focal distance, time information, and image size, type and size of the image file. The position infor- 

30 mation includes east longitude, north latitude and elevation (e.g. 137° 55' 10" of east longitude: 34° 34' 30" of north 
latitude, and elevation 101 m, 33 cm). The camera angle includes the horizontal angle and the elevation angle (e.g. 
horizontal angle 254° clockwise, elevation angle 15°). The focal distance data includes the focal distance of the camera 
lens when the image is being filmed (e.g. 28 mm). The time information includes the time of filming (e.g. 31 January 
1997, 15:15 and 15 seconds, Japanese Standard Time). The image size of the image file includes the vertical and 

3S horizontal pixel size (e.g. 640 x 480). It also has the file type (e.g. TIFE format. 8-bit color format). It also contains the 
number of bytes in the file (e.g. 307.2 KB). The image data itself is stored, for example, in binary format. 

After storing a sight image file, the control section 8B instructs the image processing section 4 to extract profile 
lines from the sight image and to divide the sight image into a plurality of sub-domains. In the image processing section 
4, profile lines (border lines ) are extracted by differential processing based on differences in concentration inside the 

40 sight image (step 25), and the sight image is divided into sub-domains by performing labeling with the profile lines as 
boundaries (step 26). The technical terminology of "labeling" used here is a technical term used in the sub-domain 
division of images, and is different from "sight-labeling" as used in the title of the invention. As a process, the image 
is first converted into a monochrome image. Since the profiles appear at portions with sudden changes in brightness, 
the profile lines are extracted by differential processing and determining the portions at which the differential values 

is are larger than a threshold value. At this time, the line width of the profile lines should be a single pixel, and the profile 
lines should be connected. Therefore, a thinning process is performed connected lines having a line width of single 
pixel. Here, the differential process and thinning process may be adequately performed by conventionally known meth- 
ods. 

The resulting profile lines are considered to be profile lines of the sub-domains, and an operation is performed of 
so numbering the sub-domains formed by the profile lines The largest of the numbers equal to the number of sub-domains, 
and the number of pixels inside each sub-domain represents the area of the sub-domain. Fig. 9 shows an example 
wherein a sight image has been divided into a plumhiy of sub-domains. A measurement of the similarity (closeness) 
between sub-domains can be incorporated so as to perform a clustering process for organizing a plurality of sub- 
domains having similar properties into a single sub domain The clustering method may be of any existing method 
ss After the control section 8B completes the process of dividing the sight image into sub-domains, it sends header 

information for the sight image file with respect to the map information management section 5 and outputs processing 
requests to calculate the view space (step 27). As an example of a map information management section 5 there is 
a map database program. The map information management section 5 manages three-dimensional map data Although 
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two-dimensional map data is acceptable, there is no height information in this case, so that the precis.on of the transfer 
position of labeling to the actual scenery is decreased. When based on two-dimensional map data, processing is 
performed by supplementing height information. For example, in the case of two^imensional data of houses, if floor 
number information representing the number of floors the houses have is available, then the height of each house can 
be estimated by multiplying the number of floors with a constant, and three-dimensional data can be prepared based 
on the height information determined by estimates of the two-dimensional data. Even if there is no floor information, 
height information can be estimated by allocating a standard height depending on the area of the house diagram or 
the lie, and preparing three-dimensional data based on estimated height information in a similar manner. The process 
progresses by preparing three-dimensional data in this way. 

Figs 5A and 5B show examples of map data. Fig. 5A shows map information space expressed in two dimensions, 
and Fig 58 shows map information space expressed in three dimensions. With this three^iimensional map information 
space the map information management section 5 receives instructions from the control section 8B to calculate a view 
space based on header information for the sight image file (step 28). Fig. 6 shows an example of the calculation of a 
view space First. XY axes are formed in the horizontal direction, and a Z axis is formed in the vertical-direction. From 
the position information in the header information of the sight image file, the position of the viewpoint E is set inside 
the three-dimensional map information space. For example, if the point is at 1 37° 55' 1 0" of east longitude, 34 34 30 
of north latitude and an elevation of 101 m 33 cm, the coordinates corresponding thereto in the map mesh numbers 
is set Similarly, the camera angle direction is set based on the horizontal angle and elevation angle in the camera 
angle information of the header information. A focal point F is set at a point which is the focal distance in front of the 
viewpoint E along the line representing the camera angle. The view line direction vector is a unit vector originating 
from this viewpoint E along this line. Using the image size in the sight image file, the width a along the horizontal axis 
in the camera screen is set by the size in the horizontal direction and the width 6 along the vertical axis is set by the 
size in the vertical direction. The plane of horizontal a and vertical b is perpendicular to the camera angle direction w.th 
respect to the view direction vector, and is set so as to include the focal point F. The lines connecting the coordinates 
of the viewpoint E with the four comers of the camera screen are each determined, and the three-dimensional space 
formed by the four half-lines extending from the viewpoint E is considered to be the view space. F.g. 7 shows an 
example of the view space with a three-dimensional map space, wherein the three-dimensional map space is viewed 
from the XZ plane The parts surrounded by the hatched lines in Fig. 7 are a cross section on the XZ plane of the space 
belonging tot he view space. In the example of Fig. 7, buildings and a mountain are contained in the view space. 

Furthermore, the map information management section 5 determines the constructions existing in the determined 
view space For each construction, it is determined whether or not the vertices forming the three-dimensional body 
representing the construction exists inside the view space. Normally, two-dimensional map spaces are div.ded by 
means of a two-dimensional mesh having a standard size. As the method of dividing the mesh in the three-dimensional 
space the mesh is cut by standard intervals in the height direction in addition to the mesh in the vertical and horizontal 
two-dimensional directions. Thus, the space is divided into cubical unit spaces. First, each cubical unit space is analyzed 
for the existence of overlaps with the view space, and the numbers of the three-dimensional unit map spaces having 
overlaps are determined. Here, the number of the three-dimensional unit map space is similar to a mesh number The 
constructions inside three-dimensional unit map spaces having overlaps are analyzed for the existence of portions 
overlapping the view space. The lines connecting the coordinates of the vertices forming the constructions and the 
coordinates of the viewpoint are determined, and they lie within the view space if the lines have an intersection with 
the camera screen of Fig. 8. Of the plurality of vertices forming a construction, if even one of the vertices fulfills this 
condition, that construction is held to have a portion overlapping the view space. 

When constructions or parts thereof are contained inside the view space, a three-dimensional projection transfor- 
mation is performed on each construction with the camera screen as the projection plane (step 29). Here as shown 
in Fig. 8, after expressing point P in a coordinate system based on viewpoint E using the following equation (1). the 
intersection Q is determined by projecting point P onto the camera screen. 
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wherein 



point P = 
point E = 
vector L = 
point P' = 



intersection Q = 



r = 



(x 5 * y z); coordinates of vertex forming construction 

(ex.. ey, ez); coordinates of viewpoint 

(Ix, /y Iz): view direction vector (unit vector) 

(xVy\ z*)\ coordinates of point P expressed in coordinate system based on viewpoint E 

(/X2 + /y2)1/2 

(X, Y): projection of point P onto camera screen t is the focal distance. 



For the three-dimensional projection transformation, the planes formed by the vertices of each construction are 
determined. For example : if the construction is expressed by a cube, then six planes will be determined. When projection 
transforming each plane onto the camera screen, the distance from the viewpoint to the corresponding point on the 
camera surface is measured with respect to each pixel on the camera surface contained in the projected sub-domain, 
and is stored in the memory as a depth value (Z value). The depth value (Z value) is calculated for each pixel on the 
camera screen for each plane of each construction, and stored in the memory In Equation (1 ), z' represents the depth 
value (A value) from the viewpoint. 

Of the constructions which are three-dimensionally projection transformed onto the camera screen, there are con- 
structions which can be seen from the viewpoint and constructions which cannot be seen from the viewpoint. Among 
these, there is a need to determine only the constructions capable of being seen from the viewpoint, and to determine 
the planes which are on the opposite side from the viewpoint or are obstructed by other constructions. Therefore, an 
occluded plane process is performed (step 30). There are many types of methods for processing occluded planes: for 
example, a Z buffer method may be used. Other possible methods include scan line methods or ray tracing methods. 

Taking an arbitrary pixel on the camera screen, the plane having the minimum depth value with respect to that 
pixel is determined. After repetitively processing each plane of each construction in this manner, the plane which is 
closest to the viewpoint for each pixel on the camera screen remains. The plane closest to the viewpoint is found for 
each pixel on the camera screen, and since the pixels on the camera screen having a common plane closest to the 
viewpoint usually form a sub-domain, a plurality of sub-domains composed of pixels which have a common plane as 
the closest plane are formed on the camera screen. The sub-domains determined in this way are sub-domains which 
result from three-dimensional transformation of the sub-domains of the constructions as seen from the viewpoint. The 
planes which are on the opposite side from the viewpoint or are obstructed by other constructions are eliminated. 

The sub-domains formed in this way form CG image domains (step 31). 

With respect to the coordinates of the vertices of the two-dimensional forms forming the CG image sub-domains, 
the three-dimensional coordinates prior to projection transformation are determined, and the correspondence between 
these is stored in memory as link information. The link information is used to determine of which construction the two- 
dimensional sub-domain is a projection. 

Based on the line data remaining after elimination of occluded lines, the CG image is divided into sub-domains. 
Since a three-dimensional map DB is used, it is possible to correlate each sub-domain with the name of the construction 
which is the basis for the sub-domain. The divided sub-domains of the CG image are numbered in order. Fig. 10 shows 
an example wherein the CG image is divided into a plurality of sub-domains. 

After the process of dividing the CG image into sub-domains is completed, the control section 8B instructs the 
label information preparation section 6B to correlate the sub-domains of the CG image with the sub-domains of the 
sight image (step 32). The label information preparation section 6B correlates the sub-domains of the CG image with 
the sub-domains of the sight image by means of template matching (step 33. see Fig. 11). 

The sub-domains of the sight image are correlated with the sub-domains of the CG image, proceeding in order 
from the sub-domain with the smallest number (for example, number 1). While it is possible to use any conventional 
matching method for the correlations, a simple template matching method is used in this case. That is, when the ratio 
of portions which overlap between two sub-domains under comparison is greater than or equal to a constant ratio 
which has been decided as being a threshold value, they are correlated as being sub-domains relating to the same 
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construction. For example, in a first sub-domain R1 of a sight image, the coordinate values of each P|»' 
domain are taken as being (A, B). The value of the pixel at coordinates (A, B) ,s 1 for being .ns.de > the » ■ £ 
the first sub-domain S1 of the CG image, if the coordinates (A, b) are inside the subOoma.n S then the P*el va'ue 
is 1 and there is overlap, but if they are outside of S1 , then the pixel value is 0 and there is no over.ap. Thus, the ove ap 
coefficient K(A B) at the coordinates (A, B) is 1 in the case of overlap, and 0 in the case of no overlap. The overlap 
coefficients K(A, B) are determined by moving the coordinates (A, B) inside the sub<Jomain R1. Of the number N1 of 
coordinates (A. B) moved inside the sub-domain R1 , the number N2 of coordinates at which the overlap coefficient K 
^6 is 1 !s determined, and if N2/N1 is greater than or equal to a thresho.d value, the sub^oma.n R1 of the sight 
mage is determined as corresponding to the sub-domain S1 of the CG image. This correlate ,s determined or he 
f™ to last sub-domain of the sight image. As another matching method, it is possible to use an evaluation funct.on 
such that they will have the same value even if there is a slight positional misalignment in the XY direction. 

After correlating the sub-domains of the CG image with the sub-domains of the sight image, the label information 
preparation section 6B enters a process of determining the information to be overlayed for each sub^omain of the 
sighUmage and preparing it along with the position to be overlayed as label information (step 34). First the sub<Joma,ns 
o, t h rCG "age corresponding to the sub-domains of the sight image are extracted. The extracted subOomams of 
the CG image are originally obtained by three-dimensional projection transformation of a plane of a three-dimens.ona 
constfucln in the three-dimensional map space onto the camera screen. Thus, the plane of ^™e^on^ 
construction which is the basis of the three-dimensional projection transformation ,s determ.ned with he depth value 
(Z value) of the sub-domain in the CG image as the key. Based on the plane of the construction which forms the bas s, 
the three-dimensional map DB is accessed to obtain the name or descriptive information of the construe x>n , . H e r e^ 
descriptive information refers to any information associated with the construction, and may be an y/ n '°™ a ^ 
to the construction. The positional coordinates at which the name or descriptive .nformat.on .s to be overlayed are 
determined with respect to the sub-domains of the sight images. The method of determination can be any method For 
example they may be at the center of gravity of the shape forming the sub-domain. Label information is prepared from 
the name of the construction, descriptive information and transfer position coordinates. Table 1 shows an example of 
label information. 



TABLE 1 


Construction Name 


Overlay Position 


Font Size 


Mt. Fuji 


(300. 500) 


10 


Building A 


(450, 250) 


10 


Building B 


(150 ; 200) 


12 



After preparing the label information, the label information preparation section 6B sends the label information to 

C Jpon leceil^the label information, the control section 8B instructs the label information output section 7B to 
output label information to the visual device for display or the like. Here, the visual device includes any video , , m age 
display device such as a display or a head-mounted display. The names or descriptive informat.on of the construe ons 
in the label information are overlayed at positions in the sight image (step 35), and the overlayed sight image .s output 
to the video image display device (step 36). Fig. 1 2 shows an example of a sight image overlayed with label informa ioa 
After the label information output section 7B outputs the label information, the contro. section 8B ,s notified of he 
completion of output. After the control section 8B receives the output completion notification, it once again repeats the 
procedures indicated above for performing sight labeling. cwelom 
Next an example wherein the sight labeling device of the present invention is applied to a communication system 
shall be explained. Fig. 13 is a diagram showing the structure of a sight labeling system wherein the sight labe hng 
device of Fig. 2 is applied to a communication system. The sight labeling system comprises a sight labeling terminal 

40 a siqht labeling center 50 and a communication network 60. 

The sight labeling terminal 40 comprises an image acquiring section 41 for acquiring images, a position in ormat on 
acquirinq section 41 for acquiring position information while the images are being taken, a camera descriptive infor- 
mation acquiring section 43 for acquiring the camera angle, focal distance and image size while the images are being 
taken an image processing section 44 for dividing the acquired images into a plurality of sub<loma,ns, communication 
control means 45 for sending information relating to the division of the images into sub-domains, the camera . angle, 
focal distance and image size through a communication network 60 to a sight labeling center 50 and receiving .at .el 
information from the sight labeling center 50, label output means 47 for overlay ing the names or de ^scnp.iye in.o mation 
of constructions in the label information at corresponding positions in the images, and outputting the overlayed images 
to a visual device, and a terminal control section 46 for controlling the above-described sections. 
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The sight labeling center 50 comprises a communication control section 53 for receiving information relating to 
division of the images into sub-domains, the camera angle, focal distance and image size from the sight labeling 
terminal 40 through the communication network 60, and sending label information to the sight labeling terminal 40. a 
map information management section 51 for managing map information, determining the view space in the map infor- 

5 mation space based on the received values for the camera angle, focal distance and image size : and capturing the 
constructions present inside the view space, a label information preparation section 52 for correlating the sub-domains 
in the images to the captured constructions by pattern matching and preparing label information including the names 
or descriptive information of the constructions and the corresponding transfer positions, and a center control section 
54 for controlling the above-described sections. 

w The label information preparation section 52 may have the same structure as the label information preparation 

section 6B in Fig. 2. 

Next, a third embodiment of a sight labeling device according to the present invention shall be explained with 
reference to the drawings. 

Fig. 14 is a diagram showing the structure of the sight labeling device of the third embodiment of the present 

is invention and Fig. 15 is a flow chart showing the process carried out by the sight labeling device of Fig. 14. In Figs. 
14 and 15, the elements which are identical to those in Figs. 1-3 are denoted by the same reference numbers. 

As shown in Fig. 14, the sight labeling device of the present embodiment comprises an image acquiring section 
1 such as a digital camera for taking sight images, a position information acquiring section 2 such as a GPS receiver 
for recording the camera position while the images are being taken, a camera descriptive information acquiring section 

20 3 such as a three-dimensional electronic compass attached to the digital camera for recording the camera angle, focal 
distance and image size while the images are being taken, an image processing section 4 for dividing the taken images 
into a plurality of sub-domains, a map information management section 5 for managing the map information, deter- 
mining the view space in the map information space based on the acquired camera position, camera angle, focal 
distance and image size, and capturing the constructions present inside the view space, a screen position indicating 

25 section 11 capable of allowing a user to indicate a position on the screen of the visual device, for determining the 
construction among the label information corresponding to a indicated screen position when a user indicates a position 
on the screen of the visual device, determining the communication address of the communication device relating to 
the determined construction from a communication address memory section 9, and sending the communication address 
to a communication processing section 10, a label information preparation section 6C for correlating the captured 

30 constructions with the sub-domains in the image and preparing label information including the name or descriptive 
information and transfer position of the corresponding construction, a label information output section 7C for overlaying 
the names and descriptive information of constructions at positions in the image corresponding to the transfer positions 
in the acquired label information, a communication address memory section 9 storing the communication addresses 
of communication devices relating to the constructions, a communication processing section 10 receiving a commu- 

35 nicatton address and for setting up a communication path with the communication address, a target condition desig- 
nating section 12 for allowing the user to designate target conditions, and a control section 8C for controlling these 
sections 1 -7C and 9-1 2. 

Next, the operations of the third embodiment shall be explained in detail with reference to Fig. 15. In the process 
shown in Fig. 15, the procedures which differ from those of the second embodiment shown in Fig. 2 are step 34C and 

40 the subsequent steps. When the device is activated, the control section 8C performs the procedures of steps 21-33 
as described above, to correlate the sub-domains of the CG image with the sub-domains of the sight image. 

After correlating the sub-domains of the CG image with the sub-domains of the sight image, the label information 
preparation section 6C enters a procedure of determining the information to be overlayed with respect to each sub- 
domain of the sight image, and preparing these together with the positions at which they are to be overlayed as label 

45 information (step 34C). First, in the same manner as the example described above, the sub-domain of the corresponding 
CG image is extracted with respect to each sub-domain of the sight image. The extracted sub-domain of the CG image 
is originally obtained by a three-dimensional projection transformation of a plane of a three-dimensional construction 
inside a three-dimensional map space onto the camera screen. The plane of the three-dimensional construction which 
is the basis of the three-dimensional projection transformation is determined using the depth value (Z value) of the 

so sub-domain of the CG image as a key. The link information which was prepared during the three-dimensional trans- 
formation may also be used as a key. Using the plane of the original construction, the three-dimensional map DB is 
accessed to obtain the name or descriptive information of the construction. Here, descriptive information refers to 
information associated with the construction, and may be any information which relates to the construction. Then, the 
positional coordinates at which the name or descriptive information is to be overlayed are determined with respect to 

55 the sub-domain of the sight image. The method of determination may be any method. For example, it may be the center 
of gravity of the shape formed by the sub-domain. The label information is prepared from the name or descriptive 
information of the construction and the positional coordinates (step 34C). At this time, in the present embodiment, the 
construction which is a target condition designated by the user in the target condition designating section 12 is given 
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a destination identifying flag "1" to indicate that it is a target condition. An example of label information is shown in 
Table 2. 

TABLE 2 



50 



55 



Construction Name 


Overlay Position 


Font Size 


Target Ident. Flag 


Mt. Fuji 


(300, 500) 


10 


0 


Building A 


(450, 250) 


10 


1 


Building B 


(150, 200) 


12 


0 



After completing preparation of the label information, the label information preparation section 6C sends the label 
information to the control section 8C. 

Upon receiving the label information, the control section 8C instructs the label information output section 7C to 
display the tabel information on the visual device. Here, the visual device may include video display devices such as 
a display or a head-mounted display. The label information output section 7C overlays the names or descriptive infor- 
mation of construction in the label information at the positions in the sight image (step 35C), then displays the overlayed 
sight image on the video display device (step 36C). In this case, the construction having a destination identifying flag 
"1 " is shown as matching the target conditions. 

When a user indicates a position on the screen of the visual device by means of the screen position indicating 
section 11, the control section 8C determines the construction corresponding to the indicated screen position from 
among the label information, determines the communication address of the communication device relating to the de- 
termined construction from the communication address memory section 9, and sends the communication address to 
the communication processing section 10 (step 37). The communication processing section 10 first displays the re- 
ceived communication address information on the screen of the visual device, then sets up a communication path with 
that communication address in response to a communication initiation request of the user (step 38). 

The communication address may also be displayed on the screen of the visual device by the label information 
output section 7C without sending it to the communication processing section 10. 

When the sight labeling procedure is to be performed continuously, the procedures described above are repeatedly 

performed. * . 

Fig. 16 is a diagram showing the structure of a sight labeling system wherein the sight labeling device of Fig. 14 
is applied to a communication system. The sight labeling system shown in Fig. 16 comprises a communication terminal 
40C. a communication center 50C and a communication network 60. In Fig. 16, the elements which are identical to 
those in Fig. 13 are denoted by the same reference numbers. 

The communication terminal 40C comprises an image acquiring section 41 for taking images, a position information 
acquiring section 42 for recording the camera position while the images are being taken, a camera descriptive infor- 
mation acquiring section 43 for obtaining the camera angle, focal distance and image size while the images are being 
taken, an image processing section 44 for dividing the obtained images into a plurality of sub-domains, a communication 
control section 45C for sending information relating to the division of the images into sub-domains, the camera position, 
camera angle, focal distance, image size, destination identifying information indicating that a location or building is a 
destination and information about constructions in the label information through a communication network 60 to a 
communication center 50C, and receiving label information and communication addresses from the communication 
center 50C. a label information output section 47 for overlaying the names or descriptive information of constructions 
in the label information at transfer positions in the images, outputting the overlayed images to a visual device, and 
displaying that a construction is the location or building which is the destination if the destination identifying flag in the 
label information is M 1 M , a communication processing section 48 for receiving a communication address and setting up 
a path to that communication address, a screen position indicating section 49A for enabling a user to indicate a position 
on the screen of the visual device, determining a construction in the label information corresponding to an indicated 
screen position when a user indicates a position on the screen of the visual device, outputting information on that 
construction to the communication control section 45C : and sending a communication address of the communication 
device relating to the construction received at the communication control section 45C to the communication processing 
section 48, a target condition designating section 49B for enabling a user to designate target conditions, and a terminal 
controt section 46C. 

The communication center 50C comprises a communication control section 53C for receiving information relating 
to the division of the images into sub-domains, the camera position, camera angle, focal distance, image size, desti- 
nation identifying information and information about the constructions in the label information from the communication 
terminal 40C through the communication network 60, and sending label information and communication addresses to 
the communication terminal 40C, a map information management section 51 for managing map information, determm- 
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ing a view space inside the map information space based on the received camera position, camera angle, focal distance 
and image size, and capturing constructions present inside the view space, a communication address memory section 
55 for storing the communication address of the communication device relating to each construction, and outputting 
the communication addresses of constructions indicated by the information for constructions in the label information, 
5 a label information preparation section 52C for correlating the captured constructions with respect to the sub<Jomains 
of the image by means of pattern matching, preparing label information including the names or descriptive information 
of the correlated constructions and the transfer positions, and making a destination identifying flag "1 " if the construction 
is a location or construction which is designated by the user as being a destination (if destination identifying information 
has been received), and a center control section 54C for controlling the above<Jescribed sections. 
w Next, a fourth embodiment of the present invention shall be explained with reference to the drawings Fig. 1 7 is a 

diagram showing the structure of a sight labeling device according to the fourth embodiment of the present invention, 
and Fig. 18 is a flow chart showing the process can-ied out by the sight labeling device of Fig. 17. In F.gs 1 7 nnd 18, 
the elements which are identical to those shown in Figs. 2 and 3 are denoted by the same reference numbers 

The sight labeling device of the fourth embodiment shown in Fig. 17 comprises an image acquiring section 1 such 
is as a digital camera for taking images, a position information acquiring section 2 such as a GPS receiver for recording 
the position of the image acquiring section 1 while the images are being taken, a camera descriptive infer m^non ac- 
quiring section 3 such as a three-dimensional electronic compass attached to the digital camera (or recording the 
camera angle, focal distance and image size while the image acquiring section 1 is taking images, an imnqe pi ocessmg 
section 4 for dividing the obtained images into a plurality of sub-domains, a distance information acqumnq sec; on 1 3 
20 for determining the distances (depth values) to each point in the images for each sub-domain of the images * :ih sped 
to the camera position white the images are being taken, a map information management section 5 for m.if^q;ra mnp 
information, determining a view space inside the map information space based on the obtained position .■■:.nr.:- 1 po- 
sition), camera angle, focal distance and image size, and capturing the constructions present inside the v o -dec 
a label information preparation section 6D for preparing a CG image based on constructions captured ::y - - mnp 
25 information management section 5, then determining the depth which is the distance to each point of each -u; :• n,-,.n 
in the CG image with respect to the camera position while the images are being taken, correlating the sub ao- : :v- 9 of 
the image with the sub-domains of the CG image by pattern matching according to a ratio between groups of : . ■ . icjo 
depth values for the points in each sub-domain of the CG image and groups of depth values for the points >r ■: : - soiv 
domain of the image, and preparing label information including the names or descriptive information of the ■; itocJ 
30 constructions and transfer positions, a label information output section 7 for overlaying the names or descr ;/. . -Joi 
mation of constructions inside the prepared label information at positions corresponding to the transfer pos : c:o 
the image, and outputting the overlayed image to a visual device, and a control section 8D for controlling 
1 -5, 6D : 7D and 1 3 described above. 

the distance information acquiring section 1 3 uses conventional art relating to threendimensional men s . ■ ■ \ ; • " ■ 
35 ods to determine the distances between each point in the sub<lomains of the sight images obtained t:> ■ i jo 
acquiring section 1 and the image acquiring section 1 (camera position) when the images are being hni ■:: •• : 
case, an example of a reference document relating to conventional methods of three-dimensional measur-: - • 

Journal of the Image Electronics Society, vol. 24, no. 5 S pp. 474-482 (Japanese). This publication expi-t ■ ■ 
such as light radar methods (pulsed light projection and modulated light projection), single-eye perspec: . •• 
40 stereo image methods and active stereo methods (slit light projection and pattern light projection). Fo- 
pulsed light projection method is a method wherein distances are determined by projecting a light pulse - 
the time until it is reflected and returned, and the modulated light projection method is a method wherein 
determined by projecting a light beam having an intensity which is temporally modulated by a sine wave o> 
wave and finding the phase difference with the reflected wave. The stereo image method is a method wf 
45 dimensional position is determined by applying triangulation between images obtained from a plurality ot 
slit light projection method is a method wherein distances are determined by triangulation of positions of » 
by projecting slit tight, and the pattern light projection method is a method for obtaining precise distano 
a smaller number of images by coding the measurement space with a light pattern. 

Since the present embodiment has a single image acquiring section 1, distances are determined 
so dimensional measurement method such as a light radar method wherein the distance information acqu" 
requires only a single image to be obtained. 

The control section 8D treats the image obtained from the image acquiring section 1 as the sight : 
image acquiring section 1 as the reference camera. The position/camera property/image information ; 
image acquiring section 1 through the control section 8D to the distance information acquiring section 
55 When the sight labeling device is activated, the control section 8D first sends the image acquiring so; 

information acquiring section 2 and the camera descriptive information acquiring section 3 a process imi : 
for obtaining information relating to the sight image. The position information acquiring section 1 receive-, 
from the control section 8D and gathers position information from the GPS receiver every second, hi---: 
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control section 8D (step 21). In this case, the time interval is not necessarily restricted to units of seconds, and any 
interval may be taken The camera descriptive information acquiring section 3 receives the instruction from the control 
section 8D and acquires the camera angle of the sight image recording device such as a camera while images are 
being taken as a group of horizontal angle and an elevation angle (step 22). and simultaneously acquires the focal 

5 distance if the sight image recording device has a zoom capability (step 23). The image acquiring section 1 receives 
an instruction from the control section 8D and takes sight images every second, and sends them to the control section 
8D (step 24) Since the image size is fixed for each sight image recording device, the control section 8D stores the 
image size information. The distance information acquiring section 1 3 determines the depth value, which is the distance 
from each point in the image with respect to the camera position while the images are being taken, at each point of 

w each sub-domain (step 24-1 ). The control section 8D stores the gathered information in a sight image file as shown in 
Fig 4 Subsequently, the procedures from steps 25-31 are performed in a manner similar to that of the second em- 
bodiment After the step of dividing the CG image into sub-domains at step 31 is completed, the control section 8D 
instructs the label information preparation section 6D to perform a correlation between the sub-domn.ns of the CG 
image and the sub-domains of the sight image. 

is After the CG image is prepared at the label information preparation section 6D, the depth value which .s the d.stance 

from each point in the sub-domains of the CG image with respect to the camera position while the images mo being 
taken are determined. Next, the average depth value for each sub-domain of the CG image is determ.neci b.sed on 
the depth value at each point in the sub-domain of the CG image (step 31-2). For example, suppose tho mmh sub- 
domain of a CG image is a sub-domain taken up by the image of a construction Z. At that time, it must be uciermmcd 

20 whether or not the construction Z actually exists in the sight image by comparing the distances (depth values , t:o! wcon 
the viewpoint and points in the image. Then, a plurality of sample coordinates X v X 2 , X 3 are taken .n rtxn sub- 
domain of the CG image, and the distances from the viewpoint (camera position) to the point X 1: the distnr 
viewpoint to the point X 2 . and the distance from the viewpoint to the point X 3 are calculated (see Fig. 1 PA 
having coordinates in the sight image corresponding to the coordinates of the point X v X 2 , X 3 are taken 

25 x,\ X 2 ' and X 3 \ The distance between the viewpoint and the point X^, the distance between the viewp 
point X 2 and the distance between the viewpoint and the point X 3 ' are each measured using the distance 
acquiring section 13. If the distances between the viewpoint and the points X v X 2 , X 3 are respectively iC- ^ * J3 m 
and 102 m. then the average depth value is 102 m. If the distances between the viewpoint and the points .x. x_ ..nd 
X 3 ' are respectively 99 m. 103 m and 101 m, then the average depth value is 101 m. In this case, the dis; -:--e m-c^.s 

30 0.99019. If the value is set to 0.90, then the construction Z is considered to exist in the sight image been • 
> 0.90. 

Next, the ratio (distance ratio) between the average depth value of each sub-domain of the CG im :;■.:! o 
average depth value of each sub-domain of the sight image is determined. On the other hand, the muz :t . . e - - .o 
(overlap ratio) between the two sub-domains to be compared in the sight image and the CG image is 
35 Then, the distance ratio and overlap ratio of the sub<iomains to be compared are multiplied, and the res 
pared with a preset threshold value. If greater than the threshold value, then they are correlated as being 
relating to the same construction. This correlation is performed for the sub-domains in the sight image 
the sub-domains with lower numbers (e.g., number 1) with respect to the sub-domains of the CG imnq. 
The calculation of the overlap ratio is performed as follows. For example, with respect to a first sub-dorr 
40 sight image, the coordinate values of each pixel in the sub-domain are taken to be (A, S). The values c 
the coordinates (A, B) are 1 due to being inside a sub-domain. In the first sub-domain S1 of the CG 
coordinates (A, B) are inside S1, then the pixel value will be 1 and they will overlap. If outside S1, the p 
be 0 and they will not overlap. In this way, the overlap coefficient K (A, B) at the coordinates ( A, B) arc 
be 1 if they overlap and 0 if they do not overlap. The coordinates (A, B) are moved inside the sub : 
45 determine the overlap coefficients K (A. B). Then, with respect to the number A/1 of coordinates (A. B) o. 
are moved in the sub-domain R1 , the number N2 of coordinates at which the overlap coefficient K 
determined, and the overlap ratio becomes N2/N1. 

As alternative matching methods, it is possible to use evaluation functions such that they will have ir 
even if there is a slight positional misalignment in the XY direction. 
so Thereafter, in the same manner as with the second embodiment, the label information prepare 

extracts the constructions of sub-domains in the CG image correlated with sub-domains in the sight if- 
then determines the information to be overlayed for each sub-domain of the sight image to prepare i,-.t; 
along with the positions at which they are to be overlayed (step 34). Then, upon receiving the prepared 
the control section 8D orders the label information output section 7D to have the visual device output I. ,: 
55 by display or the like. The names or descriptive information of constructions in the label information re- 
positions in the sight image (step 35), and the overlayed sight image is output to a video display dev..;-- 
Fig. 12). Upon outputting the label information, the label information output section 7D notifies the c - 
output completion. When the control section 8D receives the notification that the output has been 
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procedural steps described above are once again performed if the sight labeling process is to be performed continu- 
ously. 

Next, a modification of the fourth embodiment shall be explained with reference to Fig. 20 as a sight labeling device 
which is a fifth embodiment of the present invention. The sight labeling device of the fifth embodiment shown in Fig. 

5 20 comprises first and second image acquiring sections 1A : 1B such as digital cameras for taking first and second 
images, first and second position information acquiring sections 2A t 2B such as GPS receivers for respectively record- 
ing the positions of the first and second image acquiring sections 1 A, 1B while the images are being taken, first and 
second camera descriptive information acquiring sections 3A, 3B such as three-dimensional electronic compasses 
attached to the digital cameras for recording the camera angle, focal distance and image size while the first and second 

10 image acquiring sections 1 , 2 are taking images, an image processing section 4 for dividing the taken images into a 
plurality of sub-domains, a distance information acquiring section 1 3B for determining the distance (depth value) from 
the second image to each point in the first image for each sub-domain of the first image, a map information processing 
section 5 for managing map information, determining a view space inside the map information space based on the 
recorded position (camera position), camera angle, focal distance and image size, and capturing the constructions 

is present inside the view space, a label information preparation section 6D for correlating the constructions captured in 
the sub-domains of the image with ratios of the average depth values between the image and the CG image for each 
sub-domain by pattern matching, and preparing label information including the names or descriptive information of 
correlated constructions and the transfer positions, a labeUnformation output section 7D for overlaying the names or 
descriptive information of constructions in the prepared label information at the positions corresponding to the transfer 

20 positions in the image, and outputting the overlayed image to a visual device f and a control section 8E for controlling 
each section 1-7D described above. 

Since the present embodiment has a plurality of image acquiring sections, position information acquiring sections 
and camera descriptive information acquiring sections, the distance information -acquiring section 13B is capable of 
determining the camera position while images are being taken and each point in the sub-domains of the sight image 

25 not only be using the light radar method, but also by using the above-mentioned stereo image method, active stereo 
image method, and other methods which require plurality of acquired images. 

The control section 8E treats the first image acquiring section 1 A as the reference camera and the images (first 
images) obtained from the first image acquiring section 1 A as sight images. The second image acquiring section 1B 
is a camera which is installed for obtaining distance information, and the images (second images) from the second 

30 image acquiring section 1B which acquires the position, camera properties and images in the same manner as the 
first image acquiring section 1 A are not treated as sight images. The position, camera property and image information 
from the first and second image acquiring sections 1A, 1B are sent through the control section 8E to the distance 
information acquiring section 13B. 

When the sight labeling device is activated, the control section 8E first sends process initiation commands to the 

35 image acquiring sections 1 A : 1B, the position information acquiring sections 2A, 2B and the camera descriptive infor- 
mation acquiring sections 3A : 3B in order to acquire information relating to the sight images. The position information 
acquiring sections 1 A, 1B receive the instruction from the control section 8E : collect position information from a GPS 
receiver or the like every second, and sends these to the control section 8E. In this case, the time interval is not 
restricted to units of seconds, and may be any interval. The image acquiring sections 1 A, 1B receive the instructions 

JO from the control section, take sight images every second, and send these to the control section 8E. The camera de- 
scriptive information acquiring sections 3A, 3B receive the instructions from the control section 8E : acquire the camera 
angle of the sight image recording device such as a camera while the images are being taken as groups of the horizontal 
angle and elevation angle, and simultaneously acquire the focal distances if the sight image devices have zoom ca- 
pabilities. Since the image size is fixed for each sight image device, the control section 8E stores the image size 

•*5 information. The distance information acquiring section 1 3B determines the distances (depth values) from the second 
image to each point in the first image for each sub-domain. The control section 8E stores the collected information as 
a sight image file. The subsequent operations are the same as those of the fourth embodiment. 

The sight labeling device of the present invention may also have three or more image acquiring sections, position 
information acquiring sections and camera descriptive information acquiring sections. 

so Fig. 21 is a diagram showing the structure of a sight labeling system wherein the sight labeling device of Fig. 20 

is applied to a communication system. The sight labeling system comprises a sight labeling terminal 40E. a sight 
labeling center 50 and a communication network 60. In Fig. 21, the elements which are identical to those in Fig. 13 
are denoted by the same reference numbers. 

The sight labeling terminal 40E comprises first and second image acquiring section 41 A. 41 B for taking images. 

55 first and second position information acquiring sections 42A. 42B for recording the camera position while images are 
being taken, first and second camera descriptive information acquiring sections 43A. 43B for recording the camera 
angle, focal distance and image size while the images are being taken, an image processing section 44 for dividing 
the taken images into a plurality of sub-domains, a distance information acquiring section 491 for determining the depth 
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value at each point for each sub-domain in the images, which is the distance to each point in the image from the camera 
position while images are being taken, a communication control section 45 for sending information relating to the 
division of the images into sub-domains, the camera position, camera angle, focal distance and image size through 
the communication network 60 to the sight labeling center 50, and receiving label information from the sight labeling 
center 50. a label information output section 47 for overlaying the names or descriptive information of constructions in 
the label information at positions corresponding to the transfer positions in the images, and outputting the overlayed 
images to a visual device, and a terminal control section 46E for controlling each of the sections described above. 

The sight labeling center 50E comprises a communication control section 51 E for receiving the information relating 
to the division of the images into sub-domains, the camera position, camera angle, focal distance, image size and 
average depth value through the communication network 60 from the sight labeling terminal 40E, and sending the label 
information to the sight labeling terminal 40E, a map information management section 52 for managing the map data, 
determining the view space inside the map information space based on the received camera position, camera angle, 
focal distance and image size, and capturing the constructions present inside the view space, a label information 
preparation section 53E for preparing a CG image which is a computer graphics image based on the constructions 
captured by the map information management section 52, then determining the depth value which is the distance from 
the camera position while the images are being taken to each point for each sub<iomain of the CG image, correlating 
the sub-domains of the image with the sub-domains of the CG image by comparing the groups of depth values at the 
points in each sub-domain of the CG image with groups of .depth values at the points in each sub-domain of the image, 
determining the constructions in the correlated sub-domains, and preparing label information including the names and 
descriptive information of the constructions and transfer positions, and a center control section 54E for controlling each 
of the sections described above. 

The operations of the present system are the same as the operations of the device shown in Fig. 20. Additionally, 
the number of image acquiring sections, position information acquiring sections and camera descriptive information 
acquiring sections may be one each as with the fourth embodiment shown in Fig. 17, or there may be three or more. 



Claims 

1 . A device for sight labeling comprising: 

image acquiring means for taking images; 

position information acquiring means for acquiring a camera position while the images are being taken: 
camera descriptive information acquiring means for acquiring a camera angle, a focat distance and an image 
size while the images are being taken: 

map information management means for managing map information, determining a view space inside a map 
information space based on the acquired position, camera angle, focal distance and image size, and capturing 
constructions present inside the view space: 

label information preparation means for preparing label information including names or descriptive information 
of the constructions and their transfer positions: 

labeling information output means for overlaying the names or descriptive information from the map information 
at positions in the images corresponding to position information in the prepared label information, and output- 
ting the overlayed images to a visual device: and 
control means for controlling the means described above. 

2. A device for sight labeling in accordance with claim 1 , further comprising: 

image processing means for dividing images acquired by said image acquiring means into a plurality of sub- 
domains: wherein 

said label information preparation means con-elates the captured constructions with respect to sub-domains 
of said images, and prepares label information including the names or descriptive information of the correlated 
constructions and their transfer positions: 

said label information output means overlays the names or descriptive information in said label information at 
positions corresponding to their transfer positions in the images, and outputs the overlayed images to the 
visual device: and . 

said control means controls each means including said image acquiring means. 

3. A device for sight labeling in accordance with claim 2, wherein said label information preparation means prepares 
computer graphics (CG) images based on the captured constructions, correlates said sub-domains in said images 



16 



EP 0 867 690 A1 



with sub-domains in said CG images by pattern matching, determines the constructions of the correlated sub- 
domains, and prepares label information including the names or descriptive information of the constructions and 
their transfer positions. 

A device for sight labeling in accordance with claim 2, wherein said label information preparation means three- 
dimensional projection transforms the captured constructions onto the camera screen, prepares CG images by 
eliminating constructions which are not capable of being seen from the viewpoint, divides the CG images into sub- 
domains along profile lines of the sub-domains in the CG images, correlates said sub-domains in said images with 
sub-domains in said CG images by pattern matching, determines the constructions which are the basis of the sub- 
domains in the CG image correlated with the sub-domains of the images, and prepares label information including 
the names or descriptive information of the constructions and transfer positions. 

A system for sight labeling comprising a sight labeling terminal and a sight labeling center; 

said sight labeling terminal comprising 
image acquiring means for taking images, 

position information acquiring means for acquiring a camera position while the images are being taken, 
camera descriptive information acquiring means fer acquiring a camera angle, a focal distance and an image 
size while the images are being taken, 

image processing means for dividing the taken images into a plurality of sub-domains, 
communication control means for sending information relating to division of said images into sub-domains, 
the camera angle, the focal distance and the image size through the communication network to said sight 
labeling center, and receiving label information from said sight labeling center, 

label information output means for overlaying names or descriptive information of constructions inside said 
label information at corresponding positions in the images, and outputting the overlayed images to a visual 
device, and 

terminal control means for controlling the means described above; and said sight labeling center comprising 
communication control means for receiving information relating to division of said images into sub-domains, 
the camera angle, the focal distance and the image size through the communication network from said sight 
labeling terminal, and sending said label information to said sight labeling terminal, 
map information management means for managing map information, 

determining a view space inside map information space based on the received position, camera angle, focal 
distance and image size, and capturing constructions present inside the view space, 

label information preparation means for correlating the captured constructions with respect to said sub-do- 
mains of said images, and preparing label information including the names or descriptive information of the 
correlated constructions and their transfer positions, and 
center control means for controlling the means described above. 

A system for sight labeling in accordance with claim 5, wherein said label information preparation means prepares 
CG images which are computer graphics images based on the captured constructions, correlates said sub-domains 
in said images with sub-domains in said CG images by pattern matching, determines the constructions of the 
correlated sub-domains, and prepares label information including the names or descriptive information of the con- 
structions and their transfer positions. 

A system for sight labeling in accordance with claim 5 : wherein said label information preparation means three- 
dimensional projection transforms the captured constructions onto the camera screen, prepares CG images by 
eliminating constructions which are not capable of being seen from the viewpoint, divides the CG images into sub- 
domains along profile lines of the sub-domains in the CG images, correlates said sub-domains in said images with 
sub-domains in said CG images by pattern matching, determines the constructions which are the basis of the sub- 
domains in the CG image correlated with the sub-domains of the images, and prepares label information including 
the names or descriptive information of the constiuctions and their transfer positions. 

A device for sight labeling comprising: 

image acquiring means for taking images 

position information acquiring means for acquiring a camera position while the images are being taken 
camera descriptive information acquiring menus for acquiring a camera angle, a focal distance and *n image 
size while the images are being taken : 
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image processing means for dividing images acquired by said image acquiring means into a plurality of sub- 
domains: 

map information management means for managing map information, determining a view space inside a map 
information space based on the acquired position, camera angle, focal distance and image size, and capturing 
5 constructions present inside the view space; 

label information preparation means for correlating the captured constructions with respect to sub-domains of 
said images, and preparing label information including the names or descriptive information of the correlated 
constructions and their transfer positions: 

communication address memory means for storing communication addresses of communication devices re- 
w lating to said constructions: 

communication processing means for setting up communication paths with said communication addresses 
based on the received communication addresses; 

label information output means for overlaying the names or descriptive information from the map information 
at positions in the images corresponding to the transfer positions in the prepared label information, and out- 

is putting the overlayed images to* a visual device: 

screen position indicating means for allowing a user to indicate a position on the screen of the visual device, 
determining the construction in said label information corresponding to the indicated screen position when a 
position on the screen of said visual device is indicated by said user determining the communication address 
of the communication device relating to the determined construction from said communication address memory 

20 means: and 

control means for controlling the means described above. 

9. A sight labeling device in accordance with claim 8, wherein said communication processing means is constructed 
such that when said user indicates a position on the screen of the visual device, the received communication 

25 address information is temporarily displayed on said visual device, and the communication path with said commu- 

nication address is set up in response to a communication initiation request by said user. 

10. A device for sight labeling in accordance with claim 8, further comprising: 

30 image processing means for dividing images acquired by said image acquiring means into a plurality of sub- 

domains: wherein 

said label information preparation means con-elates the captured constructions with respect to sub-domains 
of said images, and prepares label information including the names or descriptive information of the con-elated 
constructions and their transfer positions: 
35 said label information output means overlays the names or descriptive information in said label information at 

positions corresponding to their transfer positions in the images, displays the overlayed images on the visual 
device, and displays a communication address on the visual device upon receiving a communication address: 
and 

said control means controls each means including said image acquiring means. 

A device for sight labeling in accordance with any one of claims 8-10, further comprising: 

target condition designating means for enabling a user to designate target conditions: wherein 
said label information preparation means appends target condition identifying information indicating that the 
target conditions designated by the user have been fulfilled to said label information: and 
said label information output means displays that a construction matches the target conditions when target 
condition identifying information is appended to said label information. 

A device for sight labeling in accordance with any one of claims 8-10, wherein said label information preparation 
means prepares CG images which are computer graphics images based on the captured constructions correlates 
said sub-domains in said images with sub-domnms in said CG images by pattern matching, determines the con- 
structions of the correlated sub-domains, and picpHres label information including the names or descriptive infor- 
mation of the constructions and their transfer positions 

A device for sight labeling in accordance with nny one of claims 5-10, wherein said label information prcpamtion 
means three-dimensional projection transforms the captured constructions onto the camera screen prepares CG 
images by eliminating constructions which are not capable of being seen from the viewpoint, divides the CG images 
into sub-domains along profile lines of the sub-domains in the CG images, correlates said sub-domnms in said 
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images with sub-domains in said CG images by pattern matching, determines the constructions which are the 
basis of the sub-domains in the CG image correlated with the sub-domains of the images, and prepares label 
information including the names or descriptive information of the constructions and their transfer positions. 

5 14. A system for sight labeling comprising a sight labeling terminal and a sight labeling center; 

said sight labeling terminal comprising 

image acquiring means for taking images, 
to position information acquiring means for acquiring a camera position while the images are being taken, 

camera descriptive information acquiring means for acquiring a camera angle, a focal distance and an 
image size while the images are being taken, 

image processing means for dividing the taken images into a plurality of sub-domains, 
communication control means for sending information relating to division of said images into sub-domains, 
is the camera angle, the focal distance and the image size through the communication network to said sight 

labeling center, and receiving label information and communication addresses from said sight labeling 
center, 

communication processing means for setting, up paths to said communication addresses based on the 
received communication addresses, 
20 label information output means for overlaying names or descriptive information of constructions inside 

said label information at corresponding positions in the images, and displaying the overlayed images on 
a visual device, 

screen position indicating means for allowing a user to indicate a position on the screen of the visual 
device, determining the construction in said label information corresponding to the indicated screen po- 
25 sition when a position on the screen of said visual device is indicated by said user, outputting said con- 

struction to said communication control means, and sending the communication address of the commu- 
nication device relating to said construction received by said communication control means to said com- 
munication processing means: and 

terminal control means for controlling the means described above; and said sight labeling center compris- 
30 ing 

communication control means for receiving information relating to division of said images into sub<fc>- 
mains, the camera angle, the focal distance and the image size through the communication network from 
said sight labeling terminal, and sending said label information and communication address to said sight 
labeling terminal, 

35 map information management means for managing map information, 

determining a view space inside map information space based on the received position, camera angle, 
focal distance and image size, and capturing constructions present inside the view space, 
label information preparation means for correlating the captured constructions with respect to said sub- 
domains of said images, and preparing label information including the names or descriptive information 

40 of the con-elated constructions and their transfer positions, 

communication address memory means for storing communication addresses of communication devices 
relating to said constructions, and outputting communication addresses of constructions indicated by in- 
formation of constructions in said label information, and 
center control means for controlling the means described above. 

45 

15. A system for sight labeling in accordance with claim 14, wherein said communication processing means is con- 
structed such that when said user indicates a position on the screen of the visual device, the received communi- 
cation address information is temporarily displayed on said visual device, and the communication path with said 
communication address is set up in response to a communication initiation request by said user. 

50 

16. A system for sight labeling comprising a sight labeling terminal and a sight labeling center: 

said sight labeling terminal comprising 

ss image acquiring means for taking images, 

position information acquiring means for acquiring a camera position while the images are being taken, 
camera descriptive information acquiring means for acquiring a camera angle, a focal distance and an 
image size while the images are being taken, 
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image processing means for dividing the taken images into a.plurality of sub-domains, 
communication control means for sending information relating to division of said images into sub^iomains, 
the camera angle, the focal distance and the image size through the communication network to said sight 
labeling center, and receiving label information and communication addresses from said sight labeling 
CGntsr 

label information output means for overlaying names or descriptive information of constructions inside 
said label information at corresponding positions in the images, and displaying the overlayed images on 
a visual device, and for displaying a communication address on said visual device when said communi- 
cation address is received, . . 
screen position indicating means for allowing user to indicate a position on the screen of the visual device, 
determining the construction in said label information corresponding to the indicated screen position when 
a position on the screen of said visual device is indicated by said user, outputting said construction to said 
communication control means, and sending the communication address of the communication device 
relating to said construction received by said communication control means to said label informat.on output 
means: and 

terminal control means for controlling the means described above; and 
said sight labeling center comprising 

communication control means for receiving information relating to division of said images into subOo- 
mains the camera angle, the focal distance and the image size through the communication network from 
said sight labeling terminal, and sending said label information and communication address to said sight 

labeling terminal, * ■ ^ 

map information management means for managing map information, determining a view space inside 
map information space based on the received position, camera angle, focal distance and image size, and 
capturing constructions present inside the view space, 

label information preparation means for correlating the captured constructions with respect to said sub- 
domains of said images, and preparing label information including the names or descriptive information 
of the correlated constructions and their transfer positions, 

communication memory means for storing communication addresses of communication devices relating 
to said constructions, and outputting communication addresses of constructions indicated by informat.on 
of constructions in said label information, and 
center control means for controlling the means described above. 

17. A system for sight labeling in accordance with any one of claims 14-16, said sight labeling terminal further com- 
prising: 

target condition designating means for enabling a user to designate target conditions: wherein 

said label information preparation means appends target condition identifying information indicating that the 

target conditions designated by the user have been fulfilled to said label information: and 

said label information output means displays that a construction matches the target conditions when target 

condition identifying information is appended to said label information. 

18 A system for sight labeling in accordance with any one of claims 14-1 6, wherein said label information preparation 
' means prepares CG images which are computer graphics images based on the captured constructions, correlates 

said sub-domains in said images with sub-domains in said CG images by pattern matching, determines the con- 
structions of the correlated sub-domains, and prepares label information including the names or descriptive infor- 
mation of the constructions and their transfer positions. 

19 A system for sight labeling in accordance with any one of claims 1 4-16, wherein said label information preparation 
means three-dimensional projection transforms the captured constructions onto the camera screen, prepares CG 
images by eliminating constructions which are not capable of being seen from the viewpoint, divides the CG images 
into sub-domains along profile lines of the sub-domains in the CG images, correlates said sub-domains ,n said 
images with sub-domains in said CG images by pattern matching, determines the constructions wh.ch are the 
basis of the sub-domains in the CG image correlated with the sub-domains of the images, and prepares label 
information including the names or descriptive information of the constructions and their transfer positions 



20. A device for sight labeling in accordance with claim 1 , further comprising: 
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distance information acquiring means for determining for each point in said images a depth value which is the 
distance to the points in said images from the camera position while the images are being taken by said image 
acquiring means: wherein 

said label information preparation means correlates the constructions captured by said map information man- 
agement means with respect to said images based on the depth values determined by said distance informa- 
tion acquiring means, and prepares label information including the names or descriptive information of the 
constructions and their transfer positions. 

21. A device for sight labeling in accordance with claim 2, further comprising: 

distance information acquiring means for determining for each point in said images a depth value which is the 
distance to the points in said images from the camera position while the images are being taken by said image 
acquiring means; wherein 

said label information preparation means correlates the constructions captured by said map information man- 
agement means with respect to said images based on the depth values determined by said distance informa- 
tion acquiring means, and prepares label information including the names or descriptive information of the 
constructions and their transfer positions. 

22. A device for sight labeling in accordance with claim 3, further comprising: 

distance information acquiring means for determining for each point in said images a depth value which is the 
distance to the points in said images from the camera position while the images are being taken by said image 
acquiring means: wherein 

said label information preparation means, after preparing CG images which are computer graphics images 
based on constructions captured by said map information management means, determines depth values which 
are the distances to each point in the sub-domains of the CG images from said camera position while said 
images are being taken, correlates the sub-domains of said images with the sub-domains of said CG images 
by pattern-matching using the ratios between groups of depth values for the points in each sub-domain of said 
CG images and groups of depth values for the points in each sub-domain of said images, determines the 
constructions of the correlated sub-domains, and prepares label information including the names or descriptive 
information of said constructions and their transfer positions. 

23. A device for sight labeling in accordance with claim 3, further comprising: 

distance information acquiring means for determining for each point in said images a depth value which is the 
distance to the points in said images from the camera position while the images are being taken by said image 
acquiring means: wherein 

said label information preparation means, after preparing CG images which are computer graphics images 
based on constructions captured by said map information management means, determines depth values which 
are the distances to each point in the sub-domains of the CG images from said camera position while said 
images are being taken, con-elates the sub-domains of said images with the sub-domains of said CG images 
by pattern-matching using the ratios between groups of depth values for the points in each sub-domain of said 
CG images and groups of depth values for the points in each sub-domain of said images and overlap ratios 
between the sub-domains of said CG images and the sub-domains of said images, determines the construc- 
tions of the correlated sub-domains, and prepares label information including the names or descriptive infor- 
mation of said constructions and their transfer positions. 

24. A device for sight labeling in accordance with any one of claims 20-23, wherein said label information preparation 
means three-dimensional projection transforms the captured constructions onto the camera screen, prepares CG 
images by eliminating constructions which are not capable of being viewed from the viewpoint, and divides the 
CG images into sub-domains by the profile lines of sub-domains in the CG images. 

25. A device for sight labeling in accordance with any one of claims 20-23, comprising a plurality of each of said image 
acquiring means, said position information acquiring means and said camera descriptive information means. 

26. A system for sight labeling in accordance with claim 5, wherein 

said sight labeling terminal further comprises distance information acquiring means for determining for each 
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point in said images a depth value which is the distance to the points in said images from the camera position 
while the images are being taken by said image acquiring means: 

in said sight labeling terminal, said label information output means sends information relating to division of 
said images into sub-domains : the camera angle, the focal distance, the image size and depth values through 
the communication network to said sight labeling center, and receives label information from said sight labeling 
center: and 

in said sight labeling center, said communication control means receives information relating to division of said 
images into sub-domains, the camera angle, the focal distance, the image size and depth values through the 
communication network from said sight labeling terminal, and sends said label information to said sight labeling 
terminal, and 

said label information preparation means, after preparing CG images which are computer graphics images 
based on constructions captured by said map information management means, determines depth values which 
are the distances to each point in the sub-domains of the CG images from said camera position while said 
images are being taken, correlates the sub-domains of said images with the sub-domains of said CG images 
by comparing groups of depth values for the points in each sub -domain of said CG images and groups of 
depth values for the points in each sub-domain of said images, determines the constructions of the correlated 
sub-domains, and prepares label information including the names or descriptive information of said construc- 
tions and their transfer positions. 

27. A system for sight labeling in accordance with claim 26, wherein in said sight labeling center, said label information 
preparation means, after preparing CG images which are computer graphics images based on constructions cap- 
tured by said map information management means, determines depth values which are the distances to each point 
in the sub-domains of the CG images from said camera position while said images are being taken, correlates the 
sub-domains of said images with the sub-domains of said CG images by pattern-matching using the ratios between 
groups of depth values for the points in each sub-domain of said CG images and groups of depth values for the 
points in each sub-domain of said images and overlap ratios between the sub-domains of said CG images and 
the sub-domains of said images, determines the constructions of the correlated sub<Jomains, and prepares label 
information including the names or descriptive information of said constructions and their transfer positions. 

28. A device for sight labeling in accordance with either claim 26 or 27, wherein said label information preparation 
means three-dimensional projection transforms the captured constructions onto the camera screen, prepares CG 
images by eliminating constructions which are not capable of being viewed from the viewpoint, and divides the 
CG images into sub-domains by the profile lines of sub-domains in the CG images. 

29. A device for sight labeling in accordance with either claim 26 or 27, comprising a plurality of each of said image 
acquiring means, said position information acquiring means and said camera descriptive information means. 
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